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Abstract 

The state-of-the-art error correcting codes are based on large random constructions (ran- 
dom graphs, random permutations, . . . ) and are decoded by linear-time iterative algorithms. 
Because of these features, they are remarkable examples of diluted mean-field spin glasses, 
both from the static and from the dynamic points of view. We analyze the behavior of decod- 
ing algorithms using the mapping onto statistical-physics models. This allows to understand 
the intrinsic (i.e. algorithm independent) features of this behavior. 



*UMR 8549, Unite Mixte de Recherche du Centre National de la Recherche Scientifique et de 1' Ecole Normale 
Superieure. 
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Figure 1: A schematic description of how error correcting codes work. 



1 Introduction 

Recently there has been some interest in studying "complexity phase transitions", i.e. abrupt 
changes in the computational complexity of hard combinatorial problems as some control 
parameter is varied These phenomena are thought to be somehow related to the physics 
of glassy systems, where the physical dynamics experiences a dramatic slowing down as the 
temperature is lowered ||] . 

Complexity is a central issue also in coding theory |3],|j]. Coding theory [||-[?J deals with the 
problem of communicating information reliably through an unreliable channel of communica- 
tion. This task is accomplished by making use of error correcting codes. In 1948 Shannon || 
proved that almost any error correcting code allows to communicate without errors, as long 
as the rate of transmitted information is kept below the capacity of the channel. However 
decoding is an intractable problem for almost any code. Coding theory is therefore a rich 
source of interesting computational problems. 

On the other hand it is known that error correcting codes can be mapped onto disordered 
spin models ||-13| . Remarkably there has recently been a revolution in coding theory which has 



brought to the invention of new and very powerful codes based on random constructions: turbo 



codes low density parity check codes (LDPCC) |15|,|16[, repetition accumulated codes |]17| , 
etc. As a matter of fact the equivalent spin models have been intensively studied in the last 
few years. These are diluted spin glasses, i.e. spin glasses on random (hyper)graphs [|il|-|2ljl , 
The new codes are decoded by using approximate iterative algorithms, which are closely 



related to the TAP-cavity approach to mean field spin glasses p^, 23|. We think therefore that 
a close investigation of these systems from a statistical physics point of view, having in mind 
complexity (i.e. dynamical) issues, can be of great theoretical interestQ 

Let us briefly recall the general setting of coding theory || in order to fix a few notations (cf. 
Fig. [I] for a pictorial description) . A source of information produces a stream of symbols. Let us 
assume, for instance, that the source produces unbiased random bits. The stream is partitioned 
into blocks of length A^iock- Each of the possible 2 Nhlock blocks is mapped to a codeword (i.e. 
a sequence of bits) of length N > A^iock by the encoder and transmitted through the channel. 
An error correcting code is therefore defined either as a mapping {0, lj^iock |o 5 1}^, or as 



1 The reader is urged to consult Refs. p4[-p3[ for a statistical mechanics analysis of the optimal decoding (i.e. of 
static issues). 
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a list of 2 Nhlock codewords. The rate of the code is defined as R = Nuock/N. 

Let us denote^ the transmitted codeword by x m = [x 1 / 1 , . . . , x^] T . Due to the noise, a 
different sequence of symbols x out = [x° ut , . . . , x^*] 1 is received. The decoding problem is to 
infer x m given x out , the definition of the code, and the properties of the noisy channel. 

It is useful to summarize the general picture which emerges from our work. We shall focus 
on Gallager codes (both regular and irregular). The optimal decoding strategy (maximum- 
likelihood decoding) is able to recover the transmitted message below some noise threshold: 
p < p c . Iterative, linear time, algorithms get stuck (in general) at a lower noise level, and 
are successful only for p < p^(alg.), with p^(alg.) < p c . In general the "dynamical" threshold 
Pd(&lg.) depends upon the details of the algorithm. However, it seems to be always smaller 
than some universal (although code-dependent) value pd- Moreover, some "optimal" linear- 
time algorithms are successful up to pd (i.e. p^(alg.) = pd). The universal threshold pd 
coincides with the dynamical transition (|j of the corresponding spin model. 

The plan of the paper is the following. In Section |2| we introduce low density parity check 
codes (LDPCC), focusing on Gallager's ensembles, and we describe message passing decoding 
algorithms. We briefly recall the connection between this algorithms and the TAP-cavity 
equations for mean-field spin glasses. In Sec. |3| we define a spin model which describes the 
decoding problem, and introduce the replica formalism. In Sec. || we analyze this model for a 
particular choice of the noisy channel (the binary erasure channel). In this case calculations 
can be fully explicit and the results are particularly clear. Then, in Sec. [j| we address the 
general case. Finally we draw our conclusions in Sec. |6|. The Appendices collect some details 
of our computations. 

2 Error correcting codes, decoding algorithms and 
the cavity equations 

This Section introduces the reader to some basic terminology in coding theory. In the first 
part we define some ensembles of codes, namely regular and irregular LDPCC. In the second 
one we describe a class of iterative decoding algorithms. These algorithms have a very clear 
physical interpretation, which we briefly recall. Finally we explain how these algorithms are 
analyzed in the coding theory community. This Section does not contain any original result. 
The interested reader may consult Refs. |7, 15, 23, 34] for further details. 



2.1 Encoding . . . 

Low density parity check codes are defined by assigning a binary N x M matrix H = {-Hy}, 
with Hij € {0, 1}. All the codewords are required to satisfy the constraint 

Hx = (mod 2). (2.1) 

The matrix H is called the parity check matrix and the M equations summarized in Eq. ( |2.1| ) 
are the parity check equations (or, for short, parity checks). If the matrix H has rank M (this 
is usually the case), the rate is R = 1 — M/N . 

2 We shall denote transmitted and received symbols by typographic characters, with the exception of symbols in 
{+1, —1}. In this case use the physicists notation and denote such symbols by a. When considering binary symbols 
we will often pass from the x notation to the a notation, the correspondence a = (— l) x being understood. Finally 
vectors of length N will be always denoted by underlined characters: e.g. x or g_. 
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Figure 2: The Tanner graph for the 7^(3) Hamming code. 



There exists a nice graphic representation of Eq. fl2.1| ) which is often used in the coding 
theory community: the Tanner graph representation pg , |36fl . One constructs a bipartite graph 
by associating a left-hand node to each one of the N variables, and a right-hand node to each 
one of the M parity checks. An edge is drawn between the variable node i and the parity check 
node a if and only if the variable Xj appears with a non-zero coefficient in the parity check 
equation a. 

Let us for instance consider the celebrated 7-^(3) Hamming code (one of the first examples 
in any book on coding theory). In this case we have N = 7, M = 3 and 



10 110 1 
10 10 11 
10 111 



(2.2) 



This code has 2 4 = 16 codewords = [x^\ . . . , xf^] 1 , with a € {1, ... , 16}. They are the 



solutions of the three parity check equations: xi + X4 + X5 + X7 = 0; X2 + X4 + x@ + X7 = 0; 
X3 + X5 + X6 + X7 = (mod2). The corresponding Tanner graph is drawn in Fig. ^. 

In general one considers ensembles of codes, by defining a random construction of the parity 
check matrix. One of the simplest ensembles is given by regular (k, I) Gallager codes. In this 
case one chooses the matrix EI randomly among all the N x M matrices having k non-zero 
entries per row, and I per column. The Tanner graph is therefore a random bipartite graph 
with fixed degrees k and I respectively for the parity check nodes and for the variable nodes. 
Of course this is possible only if M/N = l/k. 

Amazingly good codes |37|-[3^| where obtained by slightly more sophisticated irregular con- 
structions. In this case one assigns the distributions of the degrees of parity check nodes and 
variable nodes in the Tanner graph. We shall denote by {ct} the degree distribution of the 
check nodes and {vi} the degree distribution of the variable nodes. This means that there are 
Nvi bits of the codeword belonging to I parity checks and Nek parity checks involving k bits 
for each k and I. We shall always assume c^ = for k < 3 and vi = for / < 2 

It is useful to define the generating polynomials 



c(x) = s ^c k x k , v(x) = ^vix 1 , 



(2.3) 



k=3 
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Figure 3: The information coming from the channel must be used for decoding the 7^(3) Hamming 
code: a pictorial view. 



which satisfy the normalization condition c(l) = v(l) = 1. Moreover we define the average 
variable and check degrees I = v'{\) and k = c'(l). Particular examples of this formalism are 
the regular codes, whose generating polynomials are c(x) = x k , v{x) = x l . 

2.2 ... and decoding 

The codewords are transmitted trough a noisy channel. We assume antipodal signalling: one 
sends <7 m € {+1, —1} signals instead of x m € {0, f } through the channel (the correspondence 
being given by a = (— f) x ). At the end of the channel, a corrupted version of this signals 
is received. This means that if a m G {+1,-1} is transmitted, the value x out is received 
with probability density Q(x out |<7 in ). The information conveyed by the received signal x out is 
conveniently described by the log-likelihood^]: 



h(x out ) = - log Q( x ° Ut \ + 1 l . (2.4) 



We can represent this information by wavy lines in the Tanner graph, cf. Fig. ||. 

The decoding problem is to compute the probability for each transmitted bit af 1 to take 
the value <7j, given the structure of the code and the received message x out = [x° ut , . . . , x^ 1 *] 1 . 
This is in general an intractable problem |||4]]. Recently there has been a great interest in 
dealing with this problem using approximate message passing algorithms. 

Message passing algorithms are iterative: at each step t one keeps track of Mk messages 
from the variable nodes to the check nodes {yZL,j\ and viceversa {xf]^ a }. Messages can be 
thought to travel along the edges and computations to be executed at the nodes. A node 
computes the message to be sent along each one of the edges, using the messages received 
from the other (!) edges at the previous iteration (the variable nodes make also use of the 
log- likelihoods /i(x° ut )), cf. Fig. |4]. At some point the iteration is stopped (there exists no 



3 Notice the unconventional normalization: the factor 1/2 is inserted to make contact with the statistical mechanics 
formulation. 
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Figure 4: A graphic representation of the operations executed in a message passing algorithm. At 
the variable node % (on the left): = F (y^Li, VzLi'i hi) ■ At the check node a (on the right): 

„(*+!) _ nr T ® r {t) T {t) \ 



general stopping criterion), and a choice for the bit o~i is taken using all the incoming messages 
(plus the log- likelihood /i(x° ut )). 

The functions which define the "new" messages in terms of the "old" ones, can be chosen 
to optimize the decoder performances. A particularly interesting family is the following: 



.(*+!) 



hi+ E VaU (2-5) 

a'3i: a'j^a 



1 



y a ^i = arctann 



]| tanh C x fl,a 



(2.6) 



where we used the notation i G a whenever the bit i belongs to the parity check a. The 
messages {x\^ a } and can be rescaled in such a way to eliminate the parameter £ 

everywhere except in front of hi. Therefore Q allows to tune the importance given to the 
information contained in the received message. 

After the convergence of the above iteration one computes the a posteriori logdikelihoods 
as follows: 

Hi = hi + J2y^i- (2-7) 

The meaning of the {Hi} is analogous to the one of the {hi} (but for the fact that the Hi 
incorporate the information coming from the structure of the code): the best guess for the bit 
i is o~i = +1 or <jj = — 1 depending whether Hi > or Hi < 0. 

The most popular choice for the free parameter £ is C = 1: this algorithm has been invented 
separately by R. G. Gallager |l5| in the coding theory context (and named the sum-product 
algorithm) and by D. Pearl [EJ in the artificial intelligence context (and named the belief 
propagation algorithm). Also £ = oo is sometimes used (the max-product algorithm). 

The alerted reader will notice that the Eqs. (l2.5| )- (|2.6[) are nothing but the cavity equations 
at inverse temperature £ for a properly constructed spin model. This remark is the object of 
Refs. 

In the analysis of the above algorithm it is convenient to assume that af 1 = +1 for i = 
1, . . . , N. This assumption can be made without loss of generality if the channel is symmetric 
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(i.e. if Q(x| + 1) 
density 



Q(—x\ — 1)). With this assumption the hi are i.i.d. random variables with 



p(JO = Q(x(/OI + l)|x'(/i)|, 



(2.8) 



where x(/i) is the function which inverts Eq. (|2.4j) . In the following we shall consider two 
particular examples of noisy channels, the generalization being straightforward: 

• The binary erasure channel (BEC). In this case a bit can either be received correctly 
or erased^. There are therefore three possible outputs: {+1,-1,0}. The transition 
probability is: 



r (i 



Q(x out | + i) = { 



p) 
P 




if x out 
if x out 
if x out 



+1, 

0, 

-1, 



out 



1) 



ifx out 
p ifx out 
p) if x out 



+1, 

0, 

-1. 



(2.9) 



We get therefore the following distribution for the log-likelihoods: p(h) = (1—p) 5oo(h) + 
p5(h) (where 5oo is a Dirac delta function centered at +oo). Let us recall that the 
capacity of the BEC is given by Cbec = 1 — p: this means that a rate-i? code cannot 
assure error correction if p > 1 — R. 

The binary symmetric channel (BSC). The channel flips each bit independently with 
probability p. Namely 



Q(x° 



+ D 



(1 - p) if x out 
p ifx out 



+1, 
-1, 



Q(x° 



1) 



p ifx out 
(1 - p) if x out 



+1, 
-1. 



(2.10) 



The corresponding log-likelihood distribution is p(h) = (1—p) 5(h — ho) + p5(h + ho), 
with ho = arctanh(l — 2p). The capacity of the BSC isQ Cbsc = 1 — h(p) : a rate-i? code 
cannot correct errors if p > 5qv{R)- 

It is quite easy ]34|, ^2| to write a recursive equations for the probability distributions of 
the messages TTt(x) and Tf t (y): 



K t +i(x) 



I'll 



l-l 



Y[ d yi^t{yi) 

i=l 

^ \J<J nk 1 

=^c k k I Y[ dx i K t {xi 



dhp(h) S I x — h — yi 
fc-i 



S[y 



k=3 



c 



arctanh 



J^J tanh 



(2.11) 



(2.12) 



These equations (usually called the density evolution equations) are correct for times t <C log N 
due to the fact that the Tanner graph is locally tree-like. They allow therefore to predict 
whether, for a given ensemble of codes and noise level (recall that the noise level is hidden in 
p(h)) the algorithm is able to recover the transmitted codeword (for large N). If this is the 
case, the distributions irt(x) and nt(y) will concentrate on x = y = +oo as t — > oo. In the 
opposite case the above iteration will converge to some distribution supported on finite values 
of x and y. In Tab. [I] we report the threshold noise levels for several regular codes, obtained 



4 This is what happens, for instance, to packets in the Internet traffic. 

5 We denote by h(p) the binary entropy function h(p) = — p\og 2 p — (1 — p) log 2 (l — p)- It is useful to define its 
inverse: we denote by 5gv(R) (the so-called Gilbert- Varshamov distance) the smallest solution of h(5) = 1 — R. 
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BEC 


BSC 


(M) 


Pc 


Pd 


Pc 


Pd(C = 1) 


Pd(C = 2) 


Pd(( = oo) 


(6,3) 


0.4882 


0.4294 


0.100 


0.084 


0.078 


0.072 


(10,5) 


0.4995 


0.3416 


0.109 


0.070 


0.056 


0.046 


(14,7) 


0.5000 


0.2798 


0.109 


0.056 


0.039 


0.029 


(6,5) 


0.8333 


0.5510 


0.264 


0.139 


0.102 


0.078 



Table 1: The statical and dynamical points for several regular codes and decoding algorithms, cf. 
Eqs. (|J), (P). 



using the density evolution method, together with the thresholds for the optimal decoding 
strategy, see Ref. ]32| . 

Finally let us notice that the fixed point of the iteration fl2.11|) -( p.l2| ) is the replica sym- 
metric order parameter for the equivalent spin model. 

3 Statistical mechanics formulation and the replica 
approach 

We want to define a statistical mechanics model which describes the decoding problem. The 
probability distribution for the input codeword to be a = {p\, . . . , ctjv) conditional to the 
received message, takes the form 

1 f N ) 

P(a) = — 6m\cr\ exp < ^ /i^ > , (3.1) 

where <5e [oj = 1 if a; satisfies the parity checks encoded by the matrix H, cf . Eq. fl2,l|) , and 
<5h[£E] = otherwise. Since we assume the input codeword to be a" 1 = (+1, +1, . . . , +1), the 
hi are i.i.d. with distribution p(h). 

We modify the probability distribution ( |3.1| ) in two ways: 

1. We multiply the fields hi by a weight £. This allows us to tune the importance of the 
received message, analogously to Eqs. fl2.5|) and ( |2.6[ ). This modification was already 
considered in Ref. j3^| . Particularly important cases are C = 1 and C = 0. 

2. We relax the constraints implied by the characteristic function 5e[cr]. More precisely 
let us denote each parity check by the un-ordered set of bits positions (h, . . . ,ik) which 
appears in it. For instance the three parity checks in the Hamming code 7^(3), cf. Eq. 
(|2.2j ), are (1, 4, 5, 7), (2, 4, 6, 7), (3, 5, 6, 7). Moreover let Ofc be the set of all parity checks 
involving k bits (in the irregular ensemble the size of 0^ is Nek)- We can write explicitly 
the characteristic function 5ji[a] as follows: 

oo 

%[£] = II II 6(a il ---a ik ,+l), (3.2) 
fc=3 (i 1 ...i k )en k 

where 5(-, ■) is the Kronecker delta function. Now it is very simple to relax the constraints 
by making the substitution 5(ai 1 ■ ■ ■ Oi k , +1) — > exp{/3[o"j 1 • • • <Ji k — 1]}. 
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Summarizing the above considerations, we shall consider the statistical mechanics model de- 
fined by the Hamiltonian 



H(a) 



p i=i 



N 



Pi 



(3.3) 



k=3 (ii...j fc )eC fe 



at inverse temperature (3. 

We address this problem by the replica approach Q] The replicated partition function 
reads 



(Z n ) ~ ]JdX(a)dX{a) e 



NS[\,X] 



(3.4) 



with the action 



^[A, A] = I Y A (<?)A(<7) - = Y2 Ck Y J P(°^ ■ ■ ■ ><?£:)A(<ti) • • • A(<7fc) 



k=3 ai.-.a^ 



vi log 



1=2 



where 



(3.5) 



(3.6) 



(■)h being the average over p(h). The order parameters X(a) and X(a) are closely related, at 
least in the replica symmetric approximation, to the distribution of messages in the decoding 
algorithm ||, cf. Eqs. (j2lID , (^T|). 

In the case of the BEC an irrelevant infinite constant must be subtracted from the action 
( |3.5| ) in order to get finite results. This corresponds to taking 



T-Lbec(v) =P + (1 -p)S d 



(3.7) 



where <r = (+1, . . . , +1). 



4 Binary erasure channel: analytical and numerical 
results 

The binary erasure channel is simpler than the general case. Intuitively this happens because 
one cannot receive misleading indications concerning a bit. Nonetheless it is an important case 
both from the practical [Q] and from the theoretical point of view [34,33, |f|. 



4.1 The decoding algorithm 

Iterative decoding algorithms for irregular codes were first introduced and analyzed within 
this context |^8| . Belief propagation becomes particularly simple. Since the knowledge about 
a received bit is completely sure, the log-likelihoods {hi}, cf. Eq. (jO)), take the values 
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hi = +00 (when the bit has been received^ or hi = (when it has been erased). Analogously 

the messages {a;^ } and {yf^i\ must assume the same two values. The rules (|2.5|) , (^ 
become 



r (*+l) 



I +00 if either hi = +00 or y^_^ = +00 for some a' 3 i (with a' 7^ a )>u \\ 

I otherwise, 

y( t+1 ) — J +°° ^ X f-^a = +°° ^ or au tne i 6 a (with j /| 2^ 

Q ~* J otherwise. 



There exists an alternative formulation |38|] of the same algorithm. Consider the system 
of M linear equations ( |2.1| ) and eliminate from each equation the received variables (which 
are known for sure to be 0). You will obtain a new linear system. In some cases you may 
have eliminated all the variables of one equation, the equation is satisfied and can therefore 
be eliminated. For some of the other equations you may have eliminated all the variables 
but one. The remaining variable can be unambiguously fixed using this equation (since the 
received message is not misleading, this choice is surely correct). This allows to eliminate the 
variable from the entire linear system. This simple procedure is repeated until either all the 
variables have been fixed, or one gets stuck on a linear system such that all the remaining 
equations involve at least two variables (this is called a stopping set p5fl ). 

Let us for instance consider the linear system defined by the parity check matrix ([2.2;). 
Suppose, in a first case, that the received message was (0, *, 0, *, 0, *, 0) (meaning that the bits 
of positions 2, 4, 6 were erased). The decoding algorithm proceeds as follows: 

(4.3) 

In this case the algorithm succeeded in solving the decoding problem. Let us now see what 
happens if the received message is (*, 0, *, 0, *, 0, *): 





(4.4) 

The algorithm found a stopping set. Notice that the resulting linear system may well have a 
unique solution (although this is not the case in our example), which can be found by means of 
simple polynomial algorithms |^(| . Simply the iterative algorithm is unable to further reduce 
it. 

The analysis of this algorithm uses the density evolution equations ( 2.11] ), ( |2.12| ) and 



is greatly simplified because the messages {xf\ } and {y^L^} take only two values. Their 
distributions have the form: 

n t (x) = p t 6(x) + (1 - p t )<WaO = PtS(y) + (1 - p^S^y) , (4.5) 

where 5oo(-) is a delta function centered at +00. The parameters p and p give the fraction of 
zero messages, respectively from variables to checks and from checks to variables. Using Eqs. 
( fDIl) and (|2.12|) , we get: 



APt) - ! c'(l-Pt) (, p\ 

Pt + i=P-^, ^+ 1 = 1 "^(iT-- < 4 - 6 ) 



3 Recall that we are assuming the channel input to be a™ = +1 for i = 1, . . . , N. 
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Pt Pt 

Figure 5: The evolution of the iterative decoding algorithm on the BEC, cf. Eqs. ( |4.6|) . Here we 
consider the (6, 5) code: p t+ \ = p[l — (1 — Pt) 5 } 4 - On the left p = 0.5 < pd, on the right p = 0.6 > pd- 



The initial condition po = po = 1 converges to the perfect recovery fixed point p = p = if 
p < Pd- This corresponds to perfect decoding. For p > Pd the algorithm gets stuck on a non- 
trivial linear system: p t — > p* , pt — > p* , with < p* , p* < 1 . The two regimes are illustrated 
in Fig. ||. 

4.2 Statical transition 

In the spin model corresponding to the situation described above, we have two types of spins: 
the ones corresponding to correctly received bits, which are fixed by an infinite magnetic field 
h{ = +oo; and the ones corresponding to erased bits, on which no magnetic field acts: h{ = 0. 
We can therefore consider an effective model for the erased bits once the received ones are fixed 
to +1. This correspond somehow to what is done by the decoding algorithm: the received bits 
are set to their values in the very first step of the algorithm and remain unchanged thereafter. 

Let us consider the zero temperature limit. If the system is in equilibrium, its probability 
distribution will concentrate on zero energy configurations: the codewords. We will have 
typically A/" W ords(p) ~ 2 ArSwords ( p ) codewords compatible with the received message. Their 
entropy s W ords(p) can be computed within the replica formalism, cf. App. |A|. The result 
is 

s„ords (p,p;p) = IpO- -P) + = c(l - p) +pv(p) - L , (4.7) 

which has to be maximized with respect to the order parameters p and p. The saddle point 
equations have exactly the same form as the fixed point equations corresponding to the dy- 
namics ( [4.6D , namely p = pv'(p)/v'(l) and p = 1 — c'(l — p)/c'(l) 

The saddle point equations have two stable solutions, i.e. local maxima of the entropy 
([0|): (i) a completely ordered solution p = p = 0, with entropy s wor d s (0, 0) = (in some cases 
this solution becomes locally unstable above some noise pi oc )', (ii) (for sufficiently high noise 
level) a paramagnetic solution p*,/5* > 0. The paramagnetic solution appears at the same 
value pd of the noise above which the decoding algorithm gets stuck. 



11 




Figure 6: The phase diagram of the family of codes with generating polynomials c(x) = ax A + (1 — 
a)x 6 , v(x) = ax 2 + (1 — a)x 3 . The dashed line gives the local stability threshold for the completely 
ordered ferromagnetic phase. The continuous and dot-dashed lines refer (respectively) to the static 
and dynamic critical points p c (a) and pdipi)- 



The fixed point to which the dynamics (|4,q ) converges coincides with the statistical me- 
chanics result for However the entropy of the paramagnetic solution s wor ds(/9*, P*) is 
negative at pd and becomes positive only above a certain critical noise p c . This means that 
the linear system produced by the algorithm continues to have a unique solution below p c , 
although our linear time algorithm is unable find such a solution. 

The "dynamical" critical noise pd is the solution of the following equation 

v'(l)c'{l) 

where and p* solve the saddle point equations. The statical noise can be obtained setting 
s words(/°*; P*) = 0- Finally the completely ordered solution becomes locally unstable for 

m - cW(1) (49) 

Ploc v"(0)c"(l) ■ 1 ' 

As an example let us consider the one-parameter family of R = 1/2 codes specified by the 
following generating polynomials: c(x) = ax 4 + (1 — a)x 6 , v(x) = ax 2 + (1 — a)x 3 . This is 
an irregular code which smoothly interpolates between the regular (6, 3) and (4, 2) codes. The 
local stability threshold is given by 

^=S0k- <4l0) 

The dynamical and critical curves Pd{ot) and p c (a) are reported in Fig. ||. Notice that the 
a value where Pdip) reaches its maximum, corresponding to the best code in this family, is 
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neither nor 1. This is a simple example showing that irregular codes (0 < a < 1) are 
generally superior to regular ones (a = or a = 1 in this example). Notice also that above the 
tricritical point on ~ 0.79301412, p t ~ 0.39057724 the three curves pi oc (a), Pcip) and pd(os) 
coincide. In the following we shall study in some detail the a = case, which corresponds to 
a regular (6, 3) code, the corresponding critical and dynamical points p c and pd are given in 
Tab. |. 



4.3 Dynamical transition 

The dynamical transition is not properly described within the replica symmetric treatment 
given above. Indeed, the paramagnetic solution cannot be considered, between pd and p c , 
as a metastable state because it has negative entropy. One cannot therefore give a sensible 
interpretation of the coincidence between the critical noise for the decoding algorithm, and the 
appearance of the paramagnetic solution. 

Before embarking in the one step replica symmetry-breaking (1RSB) calculation, let us 
review some well-known facts [47,48]. Let us call m0(/3, m) the free energy of m weakly 



coupled "real" replicas times beta. This quantity can be computed in 1RSB calculation. In 
the limit [5 — > oo, with m(5 = \i fixed, we have m<p((3,m) — > fx(j)(fx). The number of metastable 
states with a given energy density e is 

MAis(e) ~ e N ^ , (4.11) 

where the complexity S(e) is the Legendre transform of the m replicas free energy: 

£(e) = /ie-MM)Ua[^00] • (4-12) 

The (zero temperature) dynamic energy and the static energy e s ard], respectively, the 
maximum and the minimum energy such that S(e) > 0. 

The static energy is obtained by solving the following equations: 

which corresponds to the usual prescription of maximizing the free energy over the replica 
symmetry breaking parameter m 43 ] . The dynamic energy is given by 



Finally, if e s = the complexity of the ground state is S(0) = — lim^oo /x^(/x). 

We weren't able to exactly compute the 1RSB free energy c/>(/i). However excellent results 



can be obtained within an "almost factorized" variational Ansatz, cf. App. A. 2. The picture 
which emerges is the following: 

• In the low noise region (p < pd), no metastable states exist. Local search algorithms 
should therefore be able to recover the erased bits. 



7 Notice that one can give (at least) three possible definitions of the dynamic energy: (i) from the solution of the 
nonequilibrium dynamics: ef, (ii) imposing the replicon eigenvalue to vanish: ; (Hi) using, as in the text, the 

complexity S(e): el . The three results coincide in the p-spin spherical fully connected model, however their equality 
in the present case is, at most, a conjecture. 
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Figure 7: The complexity E(e) for (from top to bottom) p = 0.45 (below p c ), p = 0.5, and p = 0.55 
(above p c ). 



• In the intermediate noise region (p^ < p < p c ) an exponentially large number of metastable 
states appears. They have energy densities e in the range e s < e < e^, with e s > 0. There- 
fore the transmitted codeword is still the only one compatible with the received message. 
Nonetheless a large number of extremely stable pseudo-codewords stop local algorithms. 
The number of violated parity checks in these codewords cannot be reduced by means of 
local moves. 

• Above p c we have e s = 0: a fraction of the metastable states is made of codewords. More- 
over S(0) (which gives the number of such codewords) coincides with the paramagnetic 
entropy s WO rds (p*, p*) computed in the previous Section. 

As an illustration, let us consider the (6, 3) regular code. In Fig. [7| we plot the resulting 
complexity curves £(e) for three different values of the erasure probability p. In Fig. || left 
frame, we report the static and dynamic energies e s and as functions of p. In the right frame 
we present the total complexity S to t = max e S(e) = S(e^), and the zero energy complexity 

2(0). 

4.4 Numerical results 

In order to check analytical predictions and to better illustrate the role of metastable states, 
we have run a set of Monte Carlo simulations, with Metropolis dynamics, on the Hamiltonian 
( |3,3i ) of the (6,3) regular code for the BEC. Notice that local search algorithms for the decoding 
problem have been already considered by the coding theory community |hj . 

We studied quite large codes (N = 10 4 bits), and tried to decode it (i.e. to find a ground 
state of the corresponding spin model) with the help of simulated annealing techniques fl50| . 
For each value of p, we start the simulation fixing a fraction (1 — p) of spins to o~i = +1 (this 
part will be kept fixed all along the run) . The remaining pN spins are the dynamical variables 
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Figure 8: Left-hand frame: the static and dynamic energies e s and of the metastable states 
(respectively, solid and dashed lines). Right-hand frame: the total complexity max e E(e) and the 
zero energy complexity S(0). 



we change during the annealing in order to try to satisfy all the parity checks. The energy of 
the system counts the number of unsatisfied parity checks. 

The cooling schedule has been chosen in the following way: r Monte Carlo sweeps (MCS) 
at each of the 1000 equidistant temperatures between T = 1 and T = 0. The highest tem- 
perature is such that the system very rapidly equilibrates on the paramagnetic energy ep(T). 
Typical values for r are from 1 to 10 3 . 

Notice that, for any fixed cooling schedule, the computational complexity of the simulated 
annealing method is linear in N. Then we expect it to be affected by metastable states of 
energy e<j, which are present for p > p^: the energy relaxation should be strongly reduced 
around 6d and eventually be completely blocked. 

In order to illustrate how the system relaxes during the simulated annealing we show in 
Fig. § the energy density as a function of the temperature for p = 0.4 (left) and p = 0.6 (right) 
and various cooling rates, r = 10, 10 2 ,10 3 (each data set is the average over many different 
samples) . 

For p = 0.4 < p<i the final energy strongly depends on the cooling rate and the slowest 
cooling procedure is always able to bring the system on the ground state, corresponding to the 
transmitted codeword. Decoding by simulated annealing is therefore successful. 

For p = 0.6 > pd the situation drastically changes. Below a temperature Td (marked by 
an arrow in Fig. ||, right frame) there is an almost complete stop of the energy relaxation. 
Td marks the dynamical transition and the corresponding energy e^(T^) = ep(T^) is called 
the threshold energy. The energy of threshold states still varies a little bit with temperature, 
€d(T), and the final value reached by the simulated annealing algorithm is its zero-temperature 

8 Each Monte Carlo sweep consists in N proposed spin flips. Each proposed spin flip is accepted or not accordingly 
to a standard Metropolis test. 
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Figure 9: Energy relaxation for the Hamiltonian of the (6,3) regular code during the simulated 
annealing with r MCS per temperature and 1000 equidistant temperatures in [0, 1] 



limit e^(0) = e^. Remember that, by construction, ground states of zero energy are present 
for any p value, but they become unreachable for p > pd, because they become shielded by 
metastable states of higher energy. 



We show in Fig. 10 the lowest energy reached by the simulated annealing procedure for 
different p and r values. While for p < pd all parity checks can be satisfied and the energy 
relaxes to zero in the limit of a very slow cooling, for p > Pd the simulation get stuck in a 
metastable state of finite energy, that is with a number of unsatisfied parity checks of order 
N. The agreement with the analytic prediction (dotted line) is quite good everywhere, but 
very close to pd- 

Discrepancies between analytical predictions and numerical results may be very well due 
to finite-size effects in the latter. One possible explanation for large finite-size effects near the 
dynamic critical point pd is the following. Metastable states of energy are stable under any 
local dynamic, which may flip simultaneously only a finite number of spins, and under global 
dynamics flipping no more than ujN spins simultaneously. Physical intuition (threshold states 
become more robust increasing p) imply that the function Lu(p) must monotonously increase 
for p £ [pd, 1]. Moreover, continuity reasons tell us that u>(pd) = 0. The fact that uj{p) is very 
small close to pd, together with the fact that in numerical simulations we are restricted to 
finite values of N, allow the local Monte Carlo dynamic to relax below the analytical predicted 
threshold energy. A more detailed characterization of this effect is presently under study and 
will be presented in a forthcoming publication. 



5 The general channel: analytical and numerical re- 
sults 

We considered the case of a general noisy channel using two different approaches: a finite- 
temperature and a zero-temperature approach. While the first one offers a clear connection 
with the dynamics of decoding-by-annealing algorithm, the second one gives a nice geometrical 
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Figure 10: Lowest energies reached by the simulated annealing. Errors are sample to sample 
fluctuations. 



picture of the situation. 

5.1 Finite temperature 

Suppose you received some message encoded using a Gallager code and you want to decode it, 
but no one explained to you the belief propagation algorithm, cf. Eqs. (|2.5|) , (|2.6|) . 

A physicist idea would be the following. Write the corresponding Hamiltonian H(a), see 
Eq. (|3.3| ), and run a Monte Carlo algorithm at inverse temperature (3. If you wait enough time, 
you will be able to sample the configuration a according to the Boltzmann distribution Pp{(j) oc 
e —pH(aj_ Then cool down the system adiabatically: i.e. change the temperature according to 
some schedule {/3i, • • • , } with (3k | oo, waiting enough time at each temperature for the 
system to equilibrate. 

As (3 — > oo the Boltzmann measure of the Hamiltonian ( |3.2[ ) concentrates on the codewords 
(for which the exchange term in Eq. (|3.2| ) is equal to zero). Moreover each codeword is given 
a weight which depends on its likelihood. In formulae: 

lim Pp{a) = ^P{a\^ : (5.1) 

where P (cr\x out ) is the probability for a to be the transmitted codeword, conditional to the 
received message x out , and is a normalization constant. Therefore when ^ > 1, our 

algorithm will sample a codeword with probability proportional to P(a\x ont )^ . For good codes 
below the critical noise threshold p c , the likelihood P(cr\x° ut ) is strongly concentrated^ on 

9 Namely we have P(er m |x out ) = 1 — 0(e~ aN ). This happens because there is a minimum O(N) Hamming distance 
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the correct input codeword. Therefore the system will spend most of its time on the correct 
codeword as soon as f3 3> 1 and ( > 1 (for ( < 1, p c has a non-trivial dependence on £, cf. 
Ref. H). 

This algorithm will succeed as long as we are able to keep the system in equilibrium at all 
temperatures down to zero. If some form of ergodicity breaking is present this may take an 
exponentially (in the size N) long time. Let us suppose to spend an 0(N) computational time 
at each temperature (3i of the annealing schedule (this is what happens in Nature). We expect 
to be able to equilibrate the system only at low enough noise (let us say for p < p<i(C)), when 
the magnetic field in Eq. ( |3.3j ) is strong enough for single out a unique ergodic component. 



5.1.1 The random linear code limit 

Some intuition on the static phase diagram can be gained by looking at the k, I — > oo limit with 



rate R = 1 — l/k fixed, cf. App. B.l.l . Unhappily, in this limit the dynamic phase transition 



disappears: the decoding algorithm is always unsuccessful, as can be understood by looking at 



Eqs. (2.5)- fl2.6Q . This phenomenon is analogous to what happens in the random energy model 



(REM) pl[: the dynamic transition is usually said to occur at infinite temperature. We refer 
to Sec. |5.2.l| for further clarifications of this point. 

There exist a paramagnetic and a ferromagnetic phases, with free energy densities 

f P = -i(log(2coshC/»)>fc + i^-log(l + tanh / 9) ) (5.2) 

fF = -|(%- (5.3) 

One must be careful in computing the entropy because of the explicit dependence of the 
Hamiltonian (|3.2| ) upon the temperature. The result is that the ferromagnetic phase has zero 
entropy sf = 0, while the entropy of the paramagnetic phase is 

Sp = (log(2coshC/i))/i - (C/i.tanhC% - (5.4) 
-(1 - R) log(l + tanh/3) + (1 - R)(3{1 - tanh (3) . 

In the low-temperature, low-noise region the paramagnetic entropy sp becomes negative. This 
signals a REM-like glassy transition [O]. The spin glass free energy is obtained by maximizing 
over the RSB parameter m (with < m < 1) the following expression 

fscim) = -Ii-p^log(l + e- 2 ^ m ) - -(log(2coshmC/i)) /l . (5.5) 
pm m 



The generic phase diagram is reported in Fig. 11. At high temperature, as the noise level 
is lowered the system undergoes a paramagnetic-ferromagnetic transition and concentrates on 
the correct codeword. At low temperature an intermediate glassy phase may be present (for 
C > 1): the system concentrates on a few incorrect configurations. 



5.1.2 Theoretical dynamical line 

The existence of metastable states can be detected within the replica formalism by the so- 
called marginal stability condition. One considers the saddle point equations for the 1RSB 

between distinct codewords [fl5|. 
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Figure 11: The phase diagram for the model ( |3.2| ) in the limit k, I — > oo with R = 1 — l/k fixed. 
Here we consider R = 1/6 and ( = 1 (on the left) and 1.5 (on the right). The rightmost (i.e. noisier) 
point for which the ferromagnetic phase is globally stable is always at (3 = oo, p = 5gv(R) ~ 0.264. 
Along the dashed line the entropy of the paramagnetic phase vanishes. 



order parameter, fixing the RSB parameter m = 1, cf. App. |B|. The dynamical temperature 
Td(p) is the highest temperature for which a "non-trivial" solution of the equation exists. 
At this temperature ergodicity of the physical dynamics breaks down (at least this is what 
happens in infinite connectivity mean field models) and we are no longer able to equilibrate 
the system within an 0(1) physical time (i.e. an O(N) computational time). 

We looked for a solution of Eqs. flB.3| )-( [Bl3|) using the population dynamics algorithm 
of Ref. We checked the "non-triviality" of the solution found by considering the vari- 

ance of the distributions p(x), p{y) (more precisely of the populations which represent such 
distributions in the algorithm). 

We consider the (6, 5) regular code because it has well separated statical and dynamical 
thresholds p c and pd, cf. Tab. II The resulting dynamical line for the Hamiltonian Q3.2j ) 
with £ = 1, is reported in Fig. 1^. The dynamic temperature T^ip) drops discontinuously 
below a noise Pd(C)'- for p < Pd(() the dynamical transition disappears and the system can 
be equilibrated in linear computational time down to zero temperature. We get Pd(l) ~ 0.14, 
which is in good agreement with the coding theory results, cf. Tab. |l| 

5.1.3 Numerical experiments 

We have repeated for the BSC the same kind of simulations already presented at the end of 



Sec. 4.4 for the BEC. 



We have run a set of simulated annealings for the Hamiltonian of the (6,5) regular 
code. System size is N = 12000 and the cooling rates are the same as for the BEC, the only 
difference being the starting and the ending temperatures, which are now T = 1.2 and T = 0.2 
(plus a quench from T = 0.2 to T = at the end of each cooling). This should not have any 
relevant effect because 0.2 -C Td ~ 0.6. 
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Figure 12: The dynamical phase transition for a regular (6, 5) code (cf. Eq. ( |3.2| ) with k = 6 and 
I = 5) with C = 1. 



The important difference with respect to the BEC case is that now we have no hxed 
spins, all N spins are dynamical variables subject to a random external field of intensity 



h = (l//3)arctanh(l - 2p), cf. Eq. (|3j) 



Also here, as in the case of the BEC, the energy relaxation for p > Pd undergoes a drastic 
arrest when the temperature is reduced below the dynamical transition at T^, see Fig. 

Unfortunately, in this case, we are not able to calculate analytically the threshold energy 
ed(0), but only the dynamical critical temperature Td and then the threshold energy at the 
transition e^(Td) which is higher than e^O). The difference Ae = e^(T^) — e^O) is usually not 
very large (see e.g. the BEC case), but it becomes apparent when p is decreased towards pd- 
Indeed for p = 0.25 (Fig. [ll| left) the Metropolis dynamics is still able to relax the system for 
temperatures below Td and then it reaches an energy well below ed(Td). On the other hand 
for p = 0.5 (Fig. [h^ right), where Ae is small the relaxation below Td is almost absent and the 
analytic prediction is much more accurate. Notice that for this case we have run a still longer 
annealing with r = 10 4 : the asymptotic energy is very close to that for r = 10 3 and hardly 
distinguishable from the analytical prediction. 



In Fig. |FJ we report the lowest energy reached by the simulated annealing for many values 
of p and r = 10, 10 2 , 10 3 , together with the analytic calculation for the threshold energy at 
Td- This analytical value is an upper bound for the true threshold energy 6^(0) where linear 
algorithms should get stuck, but it gives very accurate predictions for large p values where Ae 
is very small. In the region of small p a more complete calculation is needed. 

5.2 Zero temperature 

This approach follows from a physical intuition that is slightly different from the one explained 
in the previous paragraphs. Once again we will formulate it algorithmically. For sake of 
simplicity we shall refer, in this Section, to the BSC. We refer to the Appendix ( |B.2j ) for more 
general formulae. 
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Figure 13: Energy relaxation for the Hamiltonian of the (6,5) regular code during the simulated 
annealing with r MCS per temperature and 1000 equidistant temperatures in [0.2,1.2]. Notice 
that, in both cases p > pd- The dot-dashed line is the theoretical prediction for the paramagnetic 
exchange energy. 
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Figure 14: Lowest energies reached by the simulated annealings. Errors are sample to sample 
fluctuations. The theoretical prediction £d(Td) is computed using the results in Fig. |TJ for Td{p). 
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The overlap between the transmitted codeword and the received message 



1 N 

r^^Ew 4 , (5.6) 



is, typically, g in ' out = 1 — 2p. Given the received message, one can work in the subspace of 
all the possible configurations which have the prescribed overlap with itf^, i.e. all the a such 
that (1/-ZV) 2^*=i o'jCJ° ut ~ (1 — 2p). Once this constraint has been imposed (for instance in 
a Kawasaki-like Monte Carlo algorithm) one can restrict himself to the exchange part of the 
Hamiltonian (3^2) H cxc h(a) = — ^2 k i k ) a h " ' a ik anc ^ a PPly the cooling strategy already 



described in the previous Section. 

Below the static transition p c there exists a unique codeword having overlap (1 — 2p) with 
the received signal. This is exactly the transmitted one a in . This means that a} n is the unique 
ground state of H^^a) in the subspace we are considering. If we are able to keep our system 
in equilibrium down to T = 0, the cooling procedure will finally yield the correct answer to the 
decoding problem. Of course, if metastable states are encountered in this process, the time 
required for keeping the system in equilibrium diverges exponentially in the size. 

We expect the number of such states to be exponentially largep 1 ]: 

MMs(e,q\p)~e NE ^\ (5.7) 

where e is the exchange energy density H exc ^(a) /N. Notice that we emphasized the dependence 
of these quantities upon the noise level p. In fact the noise level determines the statistics of the 
received message a out . The static threshold is the noise level at which an exponential number 
of codewords with the same overlap as the correct one (q = 1 — 2p) appears: X p (0, 1 — 2p) > 0. 
The dynamic transition occurs where metastable states with the same overlap begin to exist: 
S p (e, 1 — 2p) > for some e > 0. 

5.2.1 The random linear code limit 

It is quite easy to compute the complexity S p (e, q) in the limit k, I — > oo with rate R = 1 — l/k 
fixed. In particular, the zeroth order term in a large k, I expansion can be derived by elementary 
methods. 

In this limit we expect the regular (k, I) ensemble to become identical to the random linear 
code (RLC) ensemble. The RLC ensemble is defined by taking each element of the parity check 



matrix H, cf. Eq. (2.1) to be or 1 with equal probability. Distinct elements are considered 
to be statistically independent. 

Let us compute the number of configurations a having a given energy and overlap with the 
received message cr out . Given a bit sequence x^O, the probability that L out of M equations 
Hx = are violated is 

Pl,*_=( M l )2- M . (5.8) 



10 Of course this is true up to 0(N 1 / 2 ) corrections. For instance one can work in the space of configurations g_ 
such that (1 — 2p ~ S)N < 2~2 i=1 <Ji(r° ut < (1 — 2p + S)N, for some small number S. 

11 For a related calculation in a fully connected model see Ref. |52]. 
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Figure 15: Metastable states in the random linear code limit for R = 1/2: their number is expo- 
nential between the continuous and the dashed lines. It vanishes discontinuously when the dashed 
line is crossed and continuously when the continuous line is crossed. The critical and dynamical 
overlaps are related to the statical and critical noise by q Cj d = 1 — 2p Cj d. In this limit pd = and 
p c = 5 GV (l/2) « 0.110025. 



Therefore the expected number of configurations x which violate L checks and have Hamming 
distance W from the received message x out is 

7^7l = <V,^ouA,o[l - 2- M ] + ( £ ) ( M L ) 2~ M , (5.9) 

where W x out is the weight of x out , i.e. its Hamming distance from 0. Notice that, up to 
exponentially small corrections, the above expression does not depend on x out . 

Introducing the overlap q = 1 — 2W/N and the exchange energy density e = 2L/N, we get 
JJwJ, ~ 2 Nt ^ with 

S(e, q) = h[(l - q)/2] + (1 - R) h[e/2(l - R)} - (1 - R) , (5.10) 

The typical number A/^ 1 ^ of such configurations can be obtained through the usual REM 

construction: M^ P L ~ 2 Ars ( e ' <? ) when S(e, q) > and M^ P L = otherwise. 

Now we are interested in picking, among all the configurations having a given energy 
density e and overlap q, the metastable states. In analogy with the REM, this can be done 
by eliminating all the configurations such that d t T,(e,q) < 0. In other words, the number of 
metastable states is Mms^iQ) ~ 2 NT,< - t,q ^ with S(e, q) = T>(e,q) when S(e, q), d e J^(e, q) > 0, 
E(e, q) = — oo otherwise. 



In Fig. 15 we plot the region of the (e,q) plane for which T,(e,q) > 0, for R = 1/2 codes. 
Notice that, in this limit S(e, q) does not depend on the received message cr out (and, therefore, 
is independent of p). As expected we get p c = 5cv(R) an d Pd = 0. 

In order to get the first non-trivial estimate for the dynamical point pd, we must consider the 
next term in the above expansion. This correction can be obtained within the replica formalism, 
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Figure 16: Metastable states for regular (k, I) codes in a large-/c,/ expansion, at fixed rate R = 1/2. 
We consider (from bottom to top) (k,l) = (6,3), (10,5), (14,7). On the left we show the region 
where S p=0 (e, q) > 0. On the right we consider instead S p (e, 1 — 2p). 



(M) 


Pc 


Pd(l) 


(6,3) 


0.097 


0.071 


(10,5) 


0.108 


0.060 


(14,7) 


0.109 


0.049 


(6,5) 


0.264 


0.108 



Table 2: Dynamical and statical thresholds at the first nontrivial order in a large k,l expansion, cf. 
Tab. |. 



see App. B.2.1 . In Fig. [T(j we reproduce contour of the region {(e, q) : £ p (e, q) > 0} for a 
few regular codes of rate R = 1/2: (A;, I) = (6, 3), (10, 5), (14, 7). The main difference between 
these curves and the exact results, cf. Sec. |5.2.2| , is the convexity of the upper boundary of 



the H p (e,q) > region (dashed lines in Figs. 15 and 16). 

The corresponding estimates for p c and p^ are reported in Tab. 



5.2.2 The complete calculation 

The full 1RSB solution for can be obtained through the population dynamics method ]i~9| . 
Here, as in Sec. 5.1.2 , we focus on the example of the (6,5) code. In Fig. 17 we plot the 
conflgurational entropy as a function of the energy of the states along the lines of constant q, 
together with the corresponding results obtained within a simple variational approach, cf. App. 



B.2.2. The approximate treatment is in quantitative agreement with the complete calculation 
for e < ed, but predicts a value for the threshold energy which is larger than the correct one: 
e d ar € d- Here e^ ar ~ 0.127 and almost p-independent. 

Unhappily the estimate of the dynamic energy obtained from this curves is not very precise. 
Moreover, at least two more considerations prevent us from comparing these results with the 
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Figure 17: The configurational entropy versus the energy for the (6, 5) regular code. Symbols refer 
to various noise levels. From top to bottom p = 0.5, 0.4, 0.35, 0.3, 0.25, 0.2, 0.18, 0.155. Continuous 
lines give the result of a variational computation, cf. App. p.2.2 . 



ones of simulated annealing simulations, cf. Sec. 5,1.3j : (i) In our annealing experiments the 



overlap with the received message cr out is free to fluctuate; (ii) We cannot exclude the 1RSB 
solution to become unstable at low temperature. 

However the population dynamics solution give the estimate pd < 0.155. This allows us to 
confirm that the point pa = 0.139 where the decoding algorithm fails to decode, cf. Tab. [I], 
coincides with the point where the metastable states appear. 



6 Conclusion 

We studied the dynamical phase transition for a large class of diluted spin models in a random 
field, the main motivation being their correspondence with very powerful error correcting 
codes. 

In a particular case, we were able to show that the dynamic critical point coincides exactly 
with the critical noise level for an important class of decoding algorithms, cf. Sec. || and App. 
|A[ For a general model of the noisy channel, we couldn't present a completely explicit proof 
of the same statement. However, within numerical precision, we obtain identical values for the 
algorithmic and the statistical mechanics thresholds. 

It may be worth listing a few interesting problems which emerge from our work: 

• Show explicitly that the identity between statistical mechanics and algorithmic thresholds 
holds in general. From a technical point of view, this is a surprising fact because the two 
thresholds are obtained, respectively, within a replica symmetric, cf Eqs. (2.11), ( 2.12j ), 
and a one-step replica symmetry breaking calculations. 

• We considered message-passing and simulated annealing algorithms. Extend the above 
analysis to other classes of algorithm (and, eventually, to any linear time algorithm). 

• Message passing decoding algorithms get stuck because they are unable to decode some 
fraction of the received message, the "hard" bits, while they have been able to decode the 
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other ones, the "easy" bits, cf. App. A.2.1. A closer look at this heterogeneous behavior 
would be very fruitful. 



It is a pleasure to thank R. Zecchina who participated to the early stages of this work. 



A Calculations: binary erasure channel 

In this Appendix we give the details of the replica calculation for the BEC. Notice that although 
we use the regular (6, 3) code as a generic example, all the computations are presented for 
general degree distributions {c^} and {vi}. 



A.l Replica symmetric approximation 



The replica symmetric calculation is correct |32| as long as we focus on codewords (i.e. on 
zero energy configurations). The main reason is the Nishimori symmetry |]53~55| which holds 
at {3 = oo and C, = 1 for the model Q3.3] ). Therefore the replica symmetric approximation gives 
access the correct noise level p c for the statical phase transition. Although such computations 
have been already considered in Refs. [2^,3^], it is interesting to review them for the BEC, 
which allows for a cleaner physical interpretation. Moreover here we generalize the already 
published results by considering a generic irregular construction. 

We parametrize the order parameters A(<r), A (a), cf. Eq. (3.5) using the Ansatz 

X(a) = dir(x)-. r^r- , X(a) = <B(y)- — — , (A.l) 

y ' J y ' (2 cosh /3x) n v ; J vy; (2cosh/3y) ri ' v ; 

where we adopted the notation J dir(x) (• • •) = f dxir(x) (• • •). It is easy to see that the order 
parameters have the form 

n(x) = (1 -p)S oa (x) +pp{x) , n(y) = p(y) 1 (A. 2) 

where <5oo(") is a Dirac delta function at +oo, and p(x), p(y) are supported on finite effective 
fields. Physically we are distinguishing the sites which correspond to correctly received bits 
(and on which an infinite magnetic field acts) from the other ones. 
The new order parameters p(x) and p(y) satisfy 



p(x) 

p(y) 



i12 Vl1 n d p(yi) s ( x - yi - ■ ■ ■ - yi-i) i ( A - 3 ) 

1 1=2 J i=l 

f v Y[ dp(xi) S(y — — arctanh[tanh/3tanh/3xi • • • tanh /?£„]) , (A. 4) 
v=o J i=i " 

f» = f E Ckk ( k Z 1 ) P^ 1 ~ ■ (A-5) 

k k=u+l V ' 

It is useful to introduce the generating function f(x) of the coefficients {f u }'- f(x) = Y^=o fuX u . 
It is easy to show that f(x) = c'(l — p + px)/c'{l). 



where 



26 



The replica symmetric free energy is obtained by substituting the above Ansatz in Eq. 
P4>[p, p] = Ip J dp(x) j dp{y) log[l + tanh/?x tanh/?y] — (A. 6) 

J OO „ V 

— = 9v I W dp{xi) log[l + tanh/3tanh (3x\ . . . t&nh(3x u ] — 

k V =Q J i=l 

00 f u 1 1 I (\ + e" 2/3 

~PJ2 Vl n\dp( yi )log[]\(l+tanhPyi)+Yl{l-taiihl3yi)]-= log' 

/ o J 1 „■ 1 „■ — 1 ^ 



1=2 " i=l i=l i=l 

with 

ff^f>( (A.7) 

The generating function of the coefficients {g u } is given by g(x) = c(l — p + px). Notice that 
{g u } is the effective degree distribution of parity check nodes (i.e. the analogous of {c^}), once 
the received bits have been eliminated. 

Let us now consider the (3 — > 00 limit. We look for solution of the following form (to the 
leading order): 

+00 

p{x)= PqS(x-q), p{y) = P+ 5(y - 1) + p 5(y) + p-5(y + 1) . (A.8) 

q= — 00 

If we define p + = ^2 q>0 p q and p- = Ylq<oPqi ^ * s eas y ^° f° ur coupled equations for the 
four variables p± and p±: 

P+ = \[f{p+ + P-) + f(P+-P-)], (A.9) 
P- = \[f{p + + P-)-f{ P+ -p-)\, (A.10) 



P+ 



P- 



fE^ E n i PTWP-- 5n ++ n 0+ n-,l-U (ATI) 



/ n+\nn\n 

1=2 n + >n_,n 



yf> E ^Jl^^^V^-.M. (AT2) 



/ ' re+!nn'n 

Z=2 n_>n + ,n 



In these equations po should be regarded as a shorthand for 1 — p + — p_. The complete 
distribution of the fields can be reconstructed from p± using the equation below 

ft = |E^ E n\n !nJ <W+n +n_,Z-l<5„ + -„_,g • (A.13) 

Z=2 n + ,no,,n„ + " 

A. 1.1 Ferromagnetic solutions 



It is clear that Eqs. (A.9)-(A.12) admit solutions with p_ = p_ = 0. Defining p = p(l — p^ 



and p = 1 — and using Eqs. ( A.9| )-( [AT2"| ), we get 
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The energy of such a solution is always zero. This means that there exists always at least one 
codeword which is compatible with the received message (this is true by construction). 

In order to compute the entropy (and therefore the number of such codewords), the finite- 
temperature corrections to the Ansatz (|A.8j) must be computed. More precisely we write: 



p(x) =J2pqPu g (f3(x-q)), p(y)=popuo(Pv)+PiPui(P(v-l)), (A.15) 

q=0 

where u q (-) and u q (-) are normalized distributions centered in zero with a well-behaved (3 — > oo 
limit. Using this Ansatz in Eqs. ( [A.3| )-( |A~4| ) and taking the (3 — > oo limit one obtain two 
coupled equations for uq(-) and uo(-). These equations can be studied using a population 
dynamics algorithm. The outcome is u„(x) = u q (x) = 5(x). The entropy is therefore correctly 
given by the simple Ansatz (|A.8|). The result is reported in Eq. Q4,7j). 



A. 1.2 Glassy solutions 

Now we look for solutions of the saddle point equations of the form (A. 8) with p_, p_ > 0. 



Such solutions have positive energy and will correspond (at most) to metastable states. The 
energy is easily written in terms of p+, p_, p+,p~ {f>o has to be interpreted as a shorthand for 
1-P+-P-): 

e[p,p] = -2pl(p + p^ + p-p + )+ l ={c[l-p + p(p + + pj)]-c[l-p + p(p + - p-)]} + 

+2pJ>, i ' i . pl + P^p n rmm(n + ,n^, (A.16) 

1=2 n+,no,n_ T 

where the sum ^ is intended to be carried over the integers n + ,no,ra_ > such that n+ + 
no + n_ = I. 

Notice that p+ and f>- can be unambiguously eliminated from the above expression by 
making use of Eqs. dAH) , ( |Al0|) . We are then left with a function of two variables: e(/0+, p~). 



Rather than studying such a function for general degree distributions {c^} and {vi}, we shall 
focus on the regular (6, 3) case: this corresponds to using c(x) = x 6 and v(x) = x 3 in Eq. 
( [A. 16 ). We expect that the behavior found in this case is generic. 



In Fig. 18 we plot e(p+,p_) for three different values of the erasure probability p\, p2 and 
P3, with p\ < p2 < Pd < P3- It is easy to guess the qualitative behavior of e(p+,p-) as p is 
varied. For small values of p no glassy extremal point with p+,p~ > can be found. At some 
value p* < pa two such points appear: a maximum and a saddle. At the dynamical threshold 
Pd the saddle point collapses onto the p- = axis. 

This picture can be confirmed by more careful study. The two glassy solutions appear 
at p* ~ 0.3844. In Fig. ( |i~9[ ) we report the corresponding energies and their position in the 
(p + ,p_) plane as functions of p. 

Which of the two solutions is the physical one? We argue in favor of the saddle point for 
the following reasons: 

• According to the standard recipe fi3fl , free energy has to be minimized with respect to 
overlaps involving an odd number of replicas and maximized with respect to overlaps of 
an even number of replicas. In our case /?+ — p_ is the magnetization (an odd overlap) 
and p + + p- is related to the two replicas overlap. We expect therefore the physical 
solution to have a stable and an unstable directions. 
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Figure 18: The replica symmetric energy ( A.16| ) as a function of p + (vertical axis) and p_ (horizontal 
axis), for p = 0.35, 0.4 and 0.45 (from left to right). Notice that only the region p + + p_ < 1 is 
meaningful. The energy vanishes as p_ — > 0. 




Figure 19: The energy density (right-hand graph) and the position in the (p + , pJ) plane of the two 
non-trivial RS solutions as a function of the erasure probability p. The continuous lines refer to the 
saddle point and the dashed line to the maximum. 
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• Two well-known relatives of our model are the ferromagnet and the dbJ spin glass on 
random hypergraphs [20, 2^]. In this cases a symmetric solution {p + = p_) exists and 
is accepted to describe the glassy states. When regarded in the full (p + ,p-) plane this 
solution appear to be a saddle point in the spin glass model, and to have zero second 
derivative in the ferromagnetic model. 

• One can try to solve Eqs, (A.9)-(A.12) iteratively. This procedure is analogous, within 
the Ansatz ( |A.8| ), to the population dynamics algorithm, which is commonly believed to 
converge to the correct solution. The iterative procedure converges to the saddle point. 

Nonetheless the qualitative picture that one would expect from the behavior of decoding 
algorithms is quite different from the one offered by the replica symmetric solution. We expect 
metastable state to appear with positive energy at pd and the minimum among their energies 
to vanish at p c . It is therefore necessary to go beyond the replica symmetric approximation. 



A. 2 Replica symmetry breaking 

The exact computation of the 1RSB free energy is a very difficult task for a finite connectivity 
model Good results can be obtained the following variational Ansatz (see Ref. for 
the general philosophy of the variational approach) 

AO?) = {l-p)5 3A)+P f{a^)-...-f{a^) 1 (A.17) 
A(<?) = /(£ (1) )-...-/(£ (n/m) ) (A.18) 

where = (a( a ~^' m+1 , . . . , a am ). This amounts to considering a fraction the spins (namely, 
the ones with an infinite magnetic field) as frozen in the +1 state, and assuming all the other 
spins to be equivalent. In the n — > limit we get d n S[X, A] — > (p[f, f] with 



[fJ] 



^iog(E/(^))-£E^ lo g(E/» 



p_ 

m 



(A.19) 



1=2 



i/=0 



E J ^ m) fe'---^fc)/fe)---/(^ 



where a are m-components replicated spins and Notice that the energy ( A.20 ) is invariant 
under a multiplicative rescaling of f{(j) and f(cr_). We shall fix this freedom by requiring that 

E £ /(£) = E £ /(2) = 1. 
Substituting 



we obtain 

P<l>[p, p] = 



dx p(x) 



f{s)= / dyp(y) 



ip 

— log 

m 

1 



(2 cosh f3x) m 



dp(x)dp{y) (1 + tanh (3x tanh f3y) r 



(2 cosh f3y) n - 



(A.20) 



=k l ° g 



1 + e 



-2/3 



km 



v=0 

oo 



J^J dp{xi) (1 + tanh (3 tanh /3x% ■ ■ ■ tanh (3a 



i=l 
I 



(A.21) 



1=2 l J i=l 



+ tanh/%i) + - tanh/3^) 



i=i 



i=l 
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and the corresponding saddle point equations: 

l-l 



(2cosh/3x) m J f=i ( 2cosh ^) 

p(y) = q /^-ivlj; 1 y d/?(yi) 5(y - -arctanh[tanh/3tanh/3yi • • • tanh/^-i 



where f u -x = gvv/{pk), cf. Eq. (|A,q ), and 



^=1 " i=l 

(A.23) 



(A.24) 



Bi = Jt[dp{vi) 



V 



Y[(l + tanh/%) + Y\( l - tanh/%) 
i=l i=l 



A u = ff[dp(xi) [1 +tanh/3tanh/?xi • • • tanh/3x^] m . (A.25) 
^ «=i 

The constants Z and Q can be chosen to enforce the normalization condition J dp(x) = 
Jdp(y) 1. 



In the j3 — > oo limit, we adopt the Ansatz (A. 8) for p{x) and /)(?/) and keep m(3 = p fixed. 
We obtain the following free energy: 

m = ^log{l + (e-^-l)[p +/ 3_ + p_p + ]}- 
P 

-^E^j 1 + \^ - mP++P-T - (P+ - P-Y\) - (A.26) 



t/=0 

oo 



p E^ lo M E ' ^un^^ ^ e " 2Atmin (n+ ' n - } 



^ ;=2 I ra+,n ,n 



the sum ^ being restricted to the integers n+,no,n_ > such that n + + no + n- = I. The 
saddle point equations are 



P+ 

^ ,=i 

CO 



-J- 52 f^A- 1 [{ P+ + p^ 1 + (p + - p^y- 1 } , (A.27) 

.. CO 

— J2 fu-iA- 1 i( P+ + p-y- 1 - (p+ - p-y- 1 } , (a.28) 



CO 



JyE^VE <W + n 0+ n_,*-i , (A.29) 



^ + ' / z — ' " 1 z — ' n_i_'nn'n 

11 1=2 n+>n-;no ~ t ' 



^ E ^ B l" E Ar#T Pl + P0°P-- e " 2Mn+ ^ ++ n 0+ n_,-l , (A.30) 



"Llf-i ^ n + !n !n 

i=2 n_>n + ;n 



where 



K = l+ 1 -(e-^-l)[(p + + p_y-( P+ -p„y], (A.31) 
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73 \ ^ -no ~ n - — 2umin(ri + ,n_) r /a gn\ 

^— ' n + !nn!n_! 

n_|_,riQ,n_ 1 



(A.33) 



i/=l 

oo 



z 



^ \7„in-l (j jOj a w + a n - -2/imin(n + ,n_) r / a oa \ 

= 2^viiB l 2^ zririzri p + p o P- e <) n+ + no+ „_,z_i(A.34) 



Z=2 n+,no,n 



It is interesting to consider some particular asymptotics of the above results. By taking the 
limit p — > we recover the replica symmetric energy ( |A.16|) and the saddle point equations 
(|A.9|) - (|A.12| ). In the p — > oo limit we have <ft(p)p — > <ftoo(p,f>)- Notice that, from Eq. ( 4.12| ), 
we get (p, /5) = — S(0), E(0) being the zero-energy complexity. The explicit expression for 
this quantity is 

oo 

0oo(p,f>) = lp\og[l-(p + p-+P-p + )]-p^2v l log{(l-p + ) 1 + (l-p-) 1 -p l }~ 

1=2 

-=f29uio g S [ i-l[(p + +p-r-(p + -p-r]} , (a.35) 

whose minimization yields the following saddle point equations: 



P+ 



P+ 



.. oo 

— £ + p^y- 1 - ( P+ - p^y- 1 ] , (a.37) 

1 oo 

-= ^ B i' [(! - z 5 -)^ 1 - (! - P+ - P-)'" 1 ] > (A-38) 

-. oo 

-= *** B i' - - (! - P+ - /5-)'- 1 ] , (A.39) 



with 



A v = i-l[( p+ + p_y-( p+ - p _y], (a.40) 

B t = (i-p + y + (i-p_) l -(i-p + -p_) 1 . (A.41) 

We look for a solution of Eqs. ( A27)-( A. 30) which is the analytic continuation of the 
"physical" one identified in the previous Section for p = 0. Such a solution exists in some 
interval pi(p) < p < P2(p)- For p < p* no physical solution exists for any value of p. For 
p* < p < pd, = p\{p) < P2(p) and 4>{p) is a monotonously increasing function between 
P\{p) and P2(p)- A physical solution exists but we cannot associate to it any "well-behaved" 
complexity. Above p^ we have < p\{p) < P2{p) = oo and a "well-behaved" complexity 
can be computed by Legendre-transforming p<f>(pj^, cf. Eq. ( 4.12j ). The complexity S(e) is 



2 The situation around pd is more complicate than the one we described. This is an artifact of the variational 
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non-zero between e s and e^. At p = p c the static energy e s vanishes: more than one codeword 
(more precisely, about exp{A r S(0)} codewords) is consistent with the received message^. 

A. 2.1 Beyond the factorized Ansatz 

The general one-step replica symmetry breaking order parameter [18] is 



/n/m 
DQ[ P ] n 
G=l 



dp(x) 



eg' 



(2 cosh j3x) 



/n/m 
DQ[ P ] n 



dp(y) 



eg' 



(2 cosh (5y) r 



(A.42) 



The saddle point equations for functional order parameters Q[p] and Q[p] are given in the next 
Section for a general channel, cf. Eqs. (|B.3|) , flB.4|) . 

In the previous Section we used a quasi-factorized Ansatz of the form: 

Q\p] = (l-p)5\p-S oo ]+pS\p-p ], Q[p] = S[p - po] , (A.43) 

where 5[-] is a functional delta function, and S oc (x) is the ordinary Dirac delta centered at 



x = +oo. This Ansatz does not satisfy the saddle point equations (BJ3), ( |B.4| ), but yields very 
good approximate results. 

Some exact results)^ (within an 1RSB scheme) can be obtained by writing the general 
decomposition 



Q[p]=uQ s [p] + (l-u)Q a [p}, Q{p]=uQs{p] + {l-u)Q a [ 



(A.44) 



where Q s [p] and Q s [p] are concentrated on the subspace of symmetric distributions (for which 
p(x) = p{—x), p(y) = p(—y)), while Q a [p] and Q a [p] have zero weight on this subspace. Using 
this decomposition in Eqs. (|B.3| ), (B.4), we get, for the BEC, a couple of equations for u and 



u, which are identical to the replica symmetric ones, cf. Eq. ( A. 14 ). 

The meaning of this result is clear. For p > Pd the system decompose in two parts. 
There exists a core which the iterative algorithms are unable to decode, and is completely 
glassy. This part is described by the functionals Q s [p] and Q s [p\- The rest of the system 
(the peripheral region) can be decoded by the belief propagation algorithm and, physically, is 
strongly magnetized. This corresponds to the functionals Q a [p] and Q a [p] (a more detailed 
study shows that the asymmetry of p and p is, in this case, typically positive). 



approximation we adopted for computing the 1RSB free energy. Here is a sketch of what happens. At p w 0.419 a 
maximum of 4>(p), which is still defined between and piip) < oo, appears. At p « 0.424 the function (j>{p) breaks 
down in two branches: a small p (defined between and p\(p) > 0), and a large p (defined between p\(j>) and 
P2{p) < oo) continuation. This second branch has a maximum for some p* . At p w 0.42715, P2(p) — > oo. This 



threshold can be computed by studying the asymptotic problem defined by Eq. (A. 35), whose physical solution is 
the saddle point lying on the p+ + p_ = 1 line. Finally, at p = pd ~ 0.429440, the small p branch disappears. 

13 Once again, because of the variational approximation we made in computing (fi(p), we obtain e s — above 
p>p' c K 0.48697. 

14 A.M. thanks M. Mezard and R. Zecchina for fruitful suggestions on this topic ]57[|. 
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B Calculations: the general channel 



In this Appendix we give some details of the replica calculation for a general noisy channel (i.e. 
for a general distribution p(h) of the random fields). In contrast with the BEC case, cf Eqs. 
( |A.SD , the local field distributions do not have a simple form even the zero temperature limit. 
Therefore our results are mainly based on a numerical solution of the saddle point equations. 



B.l Finite temperature 

The one-step replica symmetry breaking Ansatz is given in Eqs. ( [A. 42 ). Inserting in Eq. (| 
and taking the n — > limit, we get S[X, A] = ncp[Q, Q] + 0(n 2 ), with 



<t>[QM 



i 

m 



DQ[p] / DQ[p] log / dp(x) / dp(y) [1 + tp(x)tp(y)] 



-=^E Cfc Y[DQ[pi}logl i]\dp i {xi)[l + tptp(x 1 )---tp(x k )} 7 
km k=3 J i=l [ J i=l 

~^f> jj{ D QM (logU]Jdp l (y l )¥ l+1 (^,y u ...,y l )A 

-(log cosh(C/t)) 7l + l = log(l + tp) , 
k 

where we used the shorthands tp(x) = tanh(/3x), tp = tanh(/3), and defined 

n n 

¥ n (y u . . . , y n ) = J](l + ^(j/i)) + - tp{ yi )) . 
The saddle point equations are 

Q[p] = jf^vil fdp(h) fllDQ[^]S[p-p^[h,...,Pi-i}}, 

L 1=2 J J i=l 

1 oo „fc — 1 

Q[p] = =Y,c k k \\DQ[p t \5[p-p^[ Pl , 

k k=3 J i=l 



(B.l) 



(B.2) 



i=l 



i=l 



J-l 



,Pk-l\ 



where S[. . .] denotes the functional delta function, and the p h 



(Or 



(B.3) 
(B.4) 
are defined as 



follows: 



(2cosh/?z) m 

P (k \y) 



l-l 



ndpiiVi) sf (h 
, ;=1 (2cosh/^) m { -'' J ! " • 



fc-i 



i=i 



y - - arctanh[^ • tp(x{) ■ ■ ■ tp(x k -i)} 



(B.5) 
(B.6) 



These equations can be solved numerically using the population dynamics algorithm of Ref. 



1S|]. Some outcomes of this approach are reported in Sec. 5.1.2 
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B.l.l The random linear code limit 



An alternative to this numerical approach consists in considering a regular (k, I) code and 
looking at the k, I — > oo limit with fixed rate R = 1 — l/k. The leading order results are given 
in Sec. |5.1.1| . Here we give the form of the functional order parameters in this limit. 

In the ferromagnetic and paramagnetic phases the order parameter is replica symmetric: 

(B.7) 



Qp/f[p] = / dp P/F (x)5[p - 5 X ] , Qp/ F lp] = / dp P/F (y)5[p - 5 y ] , 



where S x is a delta function centered in x. Moreover we have pp{x) = p(x), pp{y) = S(y), 
and Pf{x) = Pf(x) = §oo(x)- Using these results in Eq. (|B.1|) we get the paramagnetic and 



ferromagnetic free energies, Eqs. (5.2), (]5[ 

In the spin glass phase the functionals Q[p] and Q[p] are non-trivial, although very simple: 



Qsg[p] 



dp(h) S[p - pfl) , 



)sg[p] = S\p- psa] 



where 



P { Sg( x ) 



psg(v) 



— Y / C^5(x-(h/f3-2q + l-l) 



z( h ) 



q=0 



and 



q 



2 z-i 



I - 1 



[2cosh((h + P(2q-l + l))Y 



(B.8) 

(B.9) 
(B.10) 

(B.ll) 

Notice that the order 



Substituting in Eq. flB.rj), we get the spin glass free energy (^7 
parameters (BJ)) and (|B.10[) can be used to compute the first correction to the k, I — * oo limit. 
For an example of such a calculation we refer to App. B.2.1. 



B.2 Zero temperature 

In this Appendix we compute the number of metastable states having a fixed overlap with 
a random configuration] 1 ^] er out . The dynamical and statical thresholds for the BSC can be 
deduced from the results of this computation, cf. Sec. |5.2| . The generalization to other 
statistical models for the noisy channel is straightforward (but slightly cumbersome from the 
point of view of notation). 

In order to study the existence of metastable states, we consider the constrained partition 
function: 

N 

Z{q;a out ) = e-P H °™*&6{Nq - a° ut ai) , (B.12) 

g_ i=l 

where the received bits <7° ut are i.i.d. quenched variables: <r° ut = +1 (—1) with probability 
1 — p (respectively p). We introduce m "real" weakly coupled replicas of the system: 

/+ioo m jl 
JJ^e-^Eah- £ e-fiZ^in^^+PE^E^haoT**? . (B . 13) 

a=l 2lT {**} 

15 Notice that such states are not necessarily stable with respect to moves which change their overlap with & 
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For a general channel we should look at the likelihood rather than at the overlap. 

We make the hypothesis of symmetry among the m coupled replicas. In particular we use 
the same value of the Lagrange multiplier for all of them: h a = ho/ (3m. We are therefore led 
to compute 

4>(m; h ) = - lim - log Z m (h ; a out ) n / m , (B.14) 

n-*0 n 

where 

Z m (ho;a mt ) = £ e-^r^H^fe'J+^oHE^Ef^CV," . (B .i 5 ) 

Next we take the zero temperature limit keeping m(3 = fi fixed. With a slight abuse of notation, 
we have m(j)(m;ho) — ► fMp(fi;ho)- The entropy of metastable states, cf. Eq. (p.7[), is obtained 
as the Legendre transform of //^(//; ho): 

T, p (e,q) = fie-hoq- (i^(ji;ho), (B.16) 

with e = dfj,[fj,(j)(fj,; h )] and q = -<9 foo [/i0(/i; /i )]. 

The replica expression for c/>(/i; ho) is easily obtained by taking the zero temperature limit 



on the results of Sec. B.l. The free energy reads (for sake of simplicity we write it for a regular 



(k,l) code, the generalization is trivial by making use of Eq. flB.l| )): 

l*4>[Q, Q] = l JdQIp] J DQ\p] ]og(l + jdp{x) Jdp(y)e(-xy)[e- 2 ^^-l] 



k ( , k 



~{ /ft D ^ lo § | 1 + Jtl dPi(xi)d(-xx ■ ■ ■ Xk ) [e -^MiM,---,M) _ i] J _ 
- Y[DQ[pi] ^p CT outlog< /jJ(i / 5i(y i )exp[-2^E (7 out(y 1 ...^)] > ~ 

J i=l o-out L i=l J 

-/to, (B.17) 
where p a out = 1 — p for <r out = +1, and p a out = p for <r out = — 1 and 



E CT (yi, ...,yi) = min 



i:yia<0 2:j/;(T>0 



(B.18) 



The saddle point equations become in this limit 

,1-1 



Q[p] = yJ^J^JVu* /n^N 5[p-^L[pi,---,Pz-i]], (B.19) 

1 00 1 

Q[p] = =X)c fc fc/n^0W<5[p-P ( * ) [Pi»-»,/'fc-i]]- (B.20) 

fc fc=3 3 i=\ 



The functionals /0lL t [- • -L • •] are defined as follows: 

J-l 

:/) 1 / 



?Lt(x) = \ \\\ d h{Vi) e^-^i^ ^-(VMK ut -2/i-----2//-i), (B.21) 
^ 7 i=i 
«fc-i 

(y) = / JJdpi(xi) 5[y-sign(xi---x fc _i)min(l,|xi|,...,|xA,._i|)] . (B.22) 
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B.2.1 The random linear code limit 



Here we consider the large k, I limit for the zero temperature free energy (j)(p; ho). While the 
leading order can be obtained through elementary methods, cf. Sec. |5.2.1| , the next-to-leading 
order (which is required for obtaining a non-zero dynamic threshold) must be computed within 
the replica formalism presented in the previous Section. 

We will take advantage of the fact that it is sufficient to know the saddle point order 
parameters to the leading order, in order to compute the free energy to the next-to-leading 
order. It is easy to check that, for k, I — > oo we have 



Q\p] = (l-p)6\p- P+ ]+p6\p-p-], 
where Po (y) = (1/2)% - 1) + (1/2)% + 1) and 



hc[p] =5[p- po], 



Pa{x) 



1 ' _1 

-J2C%5(x-(ho/p,)cr-2q + l-l), 



1 

2*=T 



9=0 



I - 1 



exp{| - p(l - 1) + 2pq + h cr\} ■ 



These expressions can be obtained by taking the zero temperature limit of Eqs. 
iting thesi 

p<f>(p;h ) 



Substituting these solutions into Eq. ( B.17 ) we get 

1 + e" 2 ^ 



(l--R)log 



log(l + e- 2ho )-/i l 



-(1 - R) tanh/4(l - 2p) tanh/i ] fc + (cosh p)~ l + 
+0(tanh/ig fc ,(cosh / u)~ 2 '), 



where we defined 



2- i t! 



n=(Z+l)/2 



2n) 



2 -lyl 



=1/2 



I 

1/2 



for I odd, 



for / even. 



(B.23) 

(B.24) 

(B.25) 
( TO - 

(B.26) 



(B.27) 



For / — > oo, fi(p) ~ (2-7r^ 2 ) -1 / 2 . It is easy to check that Legendre transform of the first three 
terms of Eq. ( |B.26| ) gives the elementary result ( |5,10| ). Subsequent terms give the leading 
corrections. 



B.2.2 A variational calculation 

The zero temperature equations simplify in the limit p — > oo, corresponding to vanishing 
exchange energy. In that case, a finite value of q is obtained if the magnetic field ho is kept 
finite, and it can be proved that the relation q = tanh(/io) holds. In this limit, a direct 
inspection of the saddle point equations reveals that only the values ±(1 — 1) are possible for 
the cavity fields x, and the values ±1 for the y's. More explicitly, the order parameters Q[p] 
and Q[p] are supported on distributions of the form 

p(x)=p+S(x-l + l)+p-S(x + l-l), p{y) = p+5(y - 1) + p_% - 1) . (B.28) 
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P 

Figure 20: The region of metastability as predicted by the approximated Ansatz ( |B.28| ) for the (6,5) 
code. 



The functional order parameter Q[p], reduces to the probability distributions of a single number 
p + representing the probability of y = +1. 

A simple approximation is obtained by using ( |B.28 ) and neglecting the fluctuations of p+, 
in the spirit of the "factorized Ansatz" of pC|]. This is exact F^] for ho = 0, where our model 
reduces to the one analyzed in p3|. It can be proved that, for [i = oo and /iq 0, this 



approximation gives the same result as the k,l — > oo limit, cf. Sec. 5.2.1. For instance in the 
case of (k, I) = (6, 5) we get p™ r = 0.264 which coincides with the exact result. 

The same form for the functional order parameter can also be used as a variational approx- 
imation for fi finite, although in this case it is not justified to assume y = ±1. In Fig. 20, we 



indicate the region of the (p, e) plane such that S p (e, 1 — 2p) > 0, as obtained from this simple 
approach. 
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